Image pickup apparatus

ABSTRACT

An image pickup apparatus of the present invention includes: a photographing section that can photograph a subject from a plurality of viewpoints with parallax, and can photograph a 2D moving image of the subject obtained by photographing from at least one of the viewpoints and a 3D image of the subject obtained by photographing from the plurality of the viewpoints; a recording section that records the 2D moving image and the 3D image; a subject situation determination section that determines a timing suitable for photographing the 3D image while photographing the 2D moving image; and a photographing control section that controls the photographing section so as to photograph the 3D image when the subject situation determination section determines that the timing is suitable for photographing the 3D image.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of Japanese Application No.2009-99205 filed in Japan on Apr. 15, 2009, the contents of which areincorporated herein by this reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pickup apparatus that canphotograph a stereoscopically displayable 3D image by photographing asubject from a plurality of different viewpoints.

2. Description of the Related Art

A technique has been known of photographing the same subject from aplurality of different viewpoints to photograph a stereoscopically (3D)displayable still image (3D still image) or moving image (3D movingimage).

As a method of photographing a 3D displayable image, for example, asdisclosed in Japanese Patent No. 3722498, a method is known ofphotographing from a plurality of viewpoints while moving one imagepickup apparatus by a predetermined distance. As a method ofphotographing a 3D displayable image, a method of photographing using animage pickup apparatus including two photographing lenses separated fromeach other by a predetermined distance, or a method of photographingwith two image pickup apparatuses being separated from each other by apredetermined distance and fixed is also known.

A display device that displays a 3D image has been commonly used, and auser can easily photograph and view a 3D still image or a 3D movingimage.

If a 3D moving image is always stereoscopically displayed, the viewergets used to the stereoscopic display, and the 3D moving image cannotalways provide a stronger feeling of “being there” to the viewer than ageneral 2D moving image. Thus, performance for providing a feeling of“being there” to a viewer is also important for a 3D moving image like ageneral 2D moving image.

BRIEF SUMMARY OF THE INVENTION

An image pickup apparatus of the present invention includes: aphotographing section that can photograph a subject from a plurality ofviewpoints with parallax, and can photograph a 2D moving image of thesubject obtained by photographing from at least one of the viewpointsand a 3D image of the subject obtained by photographing from theplurality of the viewpoints; a recording section that records the 2Dmoving image and the 3D image; a subject situation determination sectionthat determines a timing suitable for photographing the 3D image whilephotographing the 2D moving image; and a photographing control sectionthat controls the photographing section so as to photograph the 3D imagewhen the subject situation determination section determines that thetiming is suitable for photographing the 3D image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view showing a front side of an image pickupapparatus;

FIG. 2 is a perspective view showing a back side of the image pickupapparatus;

FIG. 3 is a block diagram illustrating an outline configuration of theimage pickup apparatus;

FIG. 4 is a flowchart of a 3D effect determination process;

FIG. 5 is a flowchart of a climax determination process;

FIG. 6 is a flowchart of a main routine;

FIG. 7 is a flowchart of the main routine;

FIG. 8 is a flowchart of a reproduction process;

FIG. 9 illustrates an operation in photographing a 2D moving image;

FIG. 10 illustrates an operation in reproducing the 2D moving image;

FIG. 11 is a flowchart of a climax determination process in a secondembodiment;

FIG. 12 shows an operation for making climax determination effective;

FIG. 13 shows an operation for photographing a 3D image using an imagepickup apparatus of a third embodiment;

FIG. 14 is a flowchart of a main routine in the third embodiment;

FIG. 15 is a flowchart of the main routine in the third embodiment;

FIG. 16 illustrates a 3D photographing guide display;

FIG. 17 shows a state at start of 2D moving image photographing; and

FIG. 18 shows a state where a 3D image is photographed.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of an image pickup apparatus of the presentinvention will be described with reference to the drawings. In thedrawings used for the description below, components have differentscales so as to have recognizable sizes in the drawings, and the presentinvention is not limited to the numbers of the components, shapes of thecomponents, size ratios of the components, and relative positionalrelationships between the components shown in the drawings.

(First Embodiment)

An image pickup apparatus of the present embodiment is an apparatus thatcan electronically record a still image and a moving image, andphotographs the same subject from a plurality of different viewpoints toobtain a stereoscopically displayable image of the subject.

Hereinafter, a combination of still images with parallax and acombination of moving images with parallax obtained by substantiallysynchronously photographing a subject from a plurality of differentviewpoints are referred to as a 3D still image and a 3D moving image,respectively. At least one of the 3D still image and the 3D moving imageis simply referred to as a 3D image.

A still image and a moving image without parallax obtained byphotographing a subject from one viewpoint are referred to as a 2D stillimage and a 2D moving image, respectively. At least one of the 2D stillimage and the 2D moving image is simply referred to as a 2D image.

An image pickup apparatus 1 of the present embodiment includes, by wayof example, a photographing section 2 that photographs a subject fromtwo viewpoints separated from each other by a predetermined base linelength BL as shown in FIG. 1. The photographing section 2 includes aleft photographing section 2 a and a right photographing section 2 bcorresponding to the two viewpoints.

The image pickup apparatus 1 can photograph the subject from differentviewpoints with the left photographing section 2 a and the rightphotographing section 2 b to record a 3D still image of the subject.Also, the image pickup apparatus 1 can photograph the subject with oneof the left photographing section 2 a and the right photographingsection 2 b to record a 2D still image of the subject. It may be allowedthat the image pickup apparatus 1 can further record a 2D moving imageand a 3D moving image of the subject.

A configuration of the photographing section 2 is not limited as long asthe photographing section 2 can photograph a subject with parallax fromtwo different viewpoints. In the present embodiment, a photographinglens and an image pickup device such as a CCD sensor or a CMOS sensorare provided in each of the left photographing section 2 a and the rightphotographing section 2 b.

The photographing section 2 may include a single image pickup device,and a pair of photographing lenses that are provided in the leftphotographing section 2 a and the right photographing section 2 b,respectively, and form a subject image on the image pickup device, andbe configured so that a light flux from the pair of photographing lensesto the image pickup device can be switched at high speed by a movablemirror or a liquid crystal shutter. For example, if a left viewpointimage obtained via the left photographing section 2 a and a rightviewpoint image obtained via the right photographing section 2 b can beswitched and photographed every 1/120 seconds by the image pickupdevice, a 3D still image and a 3D moving image obtained by substantiallysynchronously photographing the subject from the two differentviewpoints can be recorded

First, with reference to FIGS. 1 and 2, an exterior configuration of theimage pickup apparatus 1 will be described. On a front surface of theimage pickup apparatus 1 facing the subject, the left photographingsection 2 a and the right photographing section 2 b, and also amicrophone 3 for recording voice are provided.

On a back surface of the image pickup apparatus 1, a speaker 7 foroutputting voice, and an image display section 10 for displaying andoutputting an image are provided. The speaker 7 can output, for example,voice recorded simultaneously with photographing a 2D moving image or a3D moving image.

The image display section 10 can display a 2D still image and a 2Dmoving image, and can display an image with parallax photographed fromtwo different viewpoints to display a 3D still image and a 3D movingimage. Such an image display section 10 is well known, and thus detaileddescriptions thereof will be omitted. The image display section 10 maybe configured as a so-called touch panel including a touch sensor as anoperation member on a display surface.

On upper and back surfaces of the image pickup apparatus 1, an operationmember for a user to input an operation instruction of the image pickupapparatus 1 is provided. The operation member includes a still imagephotographing switch 4, a moving image photographing switch 5, a powerswitch 6, a zoom operation switch 8, a cross switch 9, and the like.

The still image photographing switch 4 is a switch for inputting aninstruction for a photographing operation of a 2D still image or a 3Dstill image by the image pickup apparatus 1. The moving imagephotographing switch 5 is a switch for inputting instructions to startand stop a photographing operation of a 2D moving image or a 3D movingimage by the image pickup apparatus 1.

The power switch 6 is a switch for inputting an instruction to switchbetween a power-on state and a power-off state of the image pickupapparatus 1. The zoom operation switch 8 is a switch for inputting aninstruction for a change operation (zoom operation) of a focal length ofthe left photographing section 2 a and the right photographing section 2b. The cross switch 9 is a switch for inputting an instruction to changean operation mode of the image pickup apparatus 1.

In the present embodiment, the operation member includes the pluralityof push switches, but the operation member may include a lever switch, adial switch, a touch sensor, a luminance sensor, an acceleration sensor,and the like.

Next, with reference to FIG. 3, a configuration for controlling anoperation of the image pickup apparatus 1 will be described. The imagepickup apparatus 1 includes a control section 20, an operationdetermination section 11, a recording section 12, and a clock section13. The image pickup apparatus 1 includes a power supply section forsupplying power required for operations of components of the imagepickup apparatus 1, although not shown. The image pickup apparatus 1 mayinclude a communication section that can transmit and receive an imageor the like to and from an external device by wired or wirelesscommunication.

The control section 20 is a control device that includes a calculationdevice, a storage device, an input/output device, a power controldevice, and the like, and controls the operations of the components ofthe image pickup apparatus 1 based on a predetermined program. Thecontrol section 20 includes an image processing section 21, a subjectsituation determination section 22, and a photographing control section23 as components required for achieving the operation of the imagepickup apparatus 1 to be described later. The image processing section21, the subject situation determination section 22, and thephotographing control section 23 may be mounted to the control section20 as hardware or software.

The image processing section 21 performs predetermined image processingof an image photographed by the photographing section 2. In the presentembodiment, the image processing section 21 includes a face recognizingsection 24 that recognizes the number, position, size and facialexpression of faces of persons in the image by predetermined imageprocessing such as pattern matching. The image processing section 21includes a contrast calculation section (not shown) that calculates acontrast value of an image used for a focusing operation of thephotographing section 2.

The subject situation determination section 22 determines the presentsituation of the subject based on information such as a focal length anda focusing length of the left photographing section 2 a and the rightphotographing section 2 b of the photographing section 2, the number,position, size and facial expression of faces of persons in the imagerecognized by the face recognizing section 24, and voice inputted fromthe microphone 3, which will be described later in detail.

The photographing control section 23 controls the photographingoperation of the photographing section 2 based on an output result fromthe subject situation determination section 22, which will be describedlater in detail.

The operation determination section 11 determines an operationinstruction inputted by a user via the above-described operation member.A determination result by the operation determination section 11 isoutputted to the control section 20. The recording section 12 is adevice that can record information, and records the image photographedby the photographing section 2 in the present embodiment. The recordingsection 12 includes a recording medium such as a nonvolatile memory of asemiconductor or a small hard disk drive. The recording section 12 maybe provided separately from the image pickup apparatus 1, and be capableof transmitting and receiving information to and from the image pickupapparatus 1 by wired or wireless communication.

The clock section 13 records a date of the image photographed by thephotographing section 2. To the image photographed by the image pickupapparatus 1, the date read from the clock section 13 is added asphotographing date information.

Now, with reference to flowcharts in FIGS. 4 to 8, the operation of theimage pickup apparatus 1 having the above-described configuration willbe described.

First, a detailed operation of the subject situation determinationsection 22 will be described. In the present embodiment, the subjectsituation determination section 22 performs a 3D effect determinationprocess and a climax determination process.

The 3D effect determination process shown in FIG. 4 is, briefly, aprocess for determining whether a distance between the image pickupapparatus 1 and a main subject has a suitable relationship with the baseline length BL of the photographing section 2 for a stereoscopicdisplay.

For example, when the distance between the image pickup apparatus 1 andthe main subject is long, a slight difference, that is, parallax isobtained between the left viewpoint image obtained by the leftphotographing section 2 a and the right viewpoint image obtained by theright photographing section 2 b, which makes it difficult to obtain a 3Dstill image or a 3D moving image that can be effectivelystereoscopically displayed.

In the present embodiment, a suitable value LA of the distance betweenthe image pickup apparatus 1 and the main subject is 3 m by way ofexample in view of the base line length BL that can be set when theimage pickup apparatus 1 is sized to be easily carried. The value LA ispreferably determined also in view of specifications of a display devicethat displays a 3D still image or a 3D moving image.

Hereinafter, a case where the distance between the image pickupapparatus 1 and the main subject has a suitable relationship with thebase line length BL of the photographing section 2 for a stereoscopicdisplay is referred to as a 3D effect being effective, and a case wherethe distance has an unsuitable relationship is referred to as a 3Deffect being ineffective.

First, in Step S70, the face recognizing section 24 performs a facerecognizing process for recognizing the presence of a person's face inthe image obtained by the photographing section 2. Then, in Step S71, itis determined whether the person's face is present at a center of theimage. Specifically, in Step S71, it is determined whether the person isa main subject.

When it is determined in Step S71 that the person's face is present atthe center of the image, the process moves to Step S72. In Steps S72 toS74, it is determined whether a distance between the image pickupapparatus 1 and the main subject has a suitable relationship for astereoscopic display based on a size Df of the face of the person as themain subject in the image and a focal length Z of the photographing lensof the photographing section 2.

Specifically, in Step S74, when a value obtained by dividing arepresentative value Df of the size of the face of the person closest tothe center in the image, for example, an area or the number of verticalpixels of the person's face by the focal length Z of the photographinglens is larger than a predetermined value AO, it is determined in StepS75 that a 3D effect is effective. When the value obtained by dividingthe representative value Df of the size of the person's face by thefocal length Z is the predetermined value AO or lower, it is determinedin Step S78 that the 3D effect is ineffective.

In Step S74, the representative value Df of the size of the person'sface is divided by the focal length Z because when the face of theperson as the main subject in the image is large and the focal length Zis long, the distance between the image pickup apparatus 1 and theperson is long, which prevents the 3D effect from being obtained. As inthe present embodiment, the representative value Df of the size of theperson's face is divided by the focal length Z, thereby eliminating aninfluence of a change in the focal length Z of the photographing lens.

The predetermined value A0 used for determination is determined in viewof the suitable value LA of the distance between the image pickupapparatus 1 and the main subject, a size of the image pickup device ofthe photographing section 2, or the like.

On the other hand, when it is determined in Step S71 that the person'sface is not present at the center of the image, that is, when the mainsubject is not a person, the process moves to Step S76. In Steps S76 andS77, a subject brought into focus is regarded as a main subject, and itis determined whether the distance between the image pickup apparatus 1and the main subject has a suitable relationship for a stereoscopicdisplay based on the focusing length L of the photographing lens.

Specifically, in Step S77, when the focusing length L of thephotographing lens is less than the suitable value LA of the distancebetween the image pickup apparatus 1 and the main subject, it isdetermined in Step S75 that the 3D effect is effective. When thefocusing length L of the photographing lens is the suitable value LA orlonger of the distance between the image pickup apparatus 1 and the mainsubject, it is determined in Step S78 that the 3D effect is ineffective.

In the present embodiment, when the person's face is recognized at thecenter of the image, it is determined whether the 3D effect is effectiveor ineffective by the size of the person's face in the image, but it maybe determined whether the 3D effect is effective or ineffective by thefocusing length L of the photographing lens image irrespective of thepresence of a person in the image.

The climax determination process shown in FIG. 5 is, briefly, a processfor determining whether the image photographed by the photographingsection 2 shows a suitable scene for a stereoscopic display inphotographing a moving image.

For example, when the main subject is a person, an image determined thatthe 3D effect is effective in the 3D effect determination process, whichshows a scene with a person in the image having no facial expression ora scene with a person during a certain action, is sometimes unsuitablefor picking out a person's charm by a stereoscopic display.

In the present embodiment, when the main subject is a person, a timewhen the person changes his/her facial expression is regarded as a scenesuitable for a stereoscopic display, and when the main subject is not aperson, a time when surrounding voice becomes louder is regarded to be acertain exciting atmosphere occurring around a photographing place andregarded as a scene suitable for a stereoscopic display by way ofexample. In photographing a moving image, a time immediately before afinish of the photographing is regarded as a scene that a user wants tophotograph, that is, a scene to be picked out and regarded as a scenesuitable for a stereoscopic display.

Hereinafter, a case where it is determined whether the imagephotographed by the photographing section 2 shows a scene suitable for astereoscopic display is referred to as climax determination beingeffective.

First in Step S80, the face recognizing section 24 performs a facerecognizing process for recognizing the presence of a person's face inthe image obtained by the photographing section 2. Then, in Step S81, itis determined whether the person's face is present at a center of theimage. Specifically, in Step S81, it is determined whether the person isa main subject.

When it is determined in Step S81 that the person's face is present atthe center of the image, the process moves to Step S82. In Step S82, theposition, size and facial expression of the face of the person as themain subject is determined and converted into numbers.

Then, in Steps S83 to S86, it is determined whether the climaxdetermination is effective based on determination results of theposition, size and facial expression of the face of the person as themain subject in the image in the present frame and a predeterminednumber of previous frames.

Specifically, in Step S83, when there is no change in any of theposition, size and facial expression of the person's face in the image,the process moves to Step S84, and it is determined whether the facialexpression of the person is a smile. When the facial expression of theperson is a smile in Step S84, it is determined that the climaxdetermination is effective.

In Step S83, when there is a change in any of the position, size andfacial expression of the person's face in the image, the process movesto Step S85. When the change in the position, size and facial expressionof the person's face in the image reaches a predetermined threshold orless, that is, when the person's face stops, it is determined that theclimax determination is effective.

In Step S85, even when the change in the position, size and facialexpression of the person's face in the image continues, it is determinedthat the climax determination is effective if the moving imagephotographing switch 5 is operated to input an instruction to finishphotographing a moving image.

On the other hand, when it is determined in Step S81 that the person'sface is not present at the center of the image, that is, when the mainsubject is not a person, the process moves to Step S88. In Step S88, itis determined that the climax determination is effective when the voiceinputted from the microphone 3 becomes louder.

Next, with reference to FIGS. 6 and 7, a main routine of operationcontrol of the image pickup apparatus 1 will be described. The mainroutine is started by operating the power switch 6 to power on the imagepickup apparatus 1.

First in Step S01, it is determined whether the power switch 6 isoperated to input an instruction to power off the image pickup apparatus1. When it is determined that the instruction to power off the imagepickup apparatus 1 is inputted, a finish process such as moving theposition of the photographing lens to a predetermined position isperformed, and then the image pickup apparatus 1 is powered off.

When it is determined that the instruction to power off the image pickupapparatus 1 is not inputted, the process moves to Step S02, and it isdetermined whether an operation mode of the image pickup apparatus 1 isset to a photographing mode.

When it is determined in Step S02 that the operation mode is not set tothe photographing mode, it is determined in Step S03 whether theoperation mode of the image pickup apparatus 1 is set to a reproductionmode. When it is determined in Step S03 that the operation mode is notset to the reproduction mode, the presence of an abnormality in settingof the operation mode is displayed on the image display section 10, andthe process returns to Step S01. When it is determined in Step S03 thatthe operation mode is set to the reproduction mode, a reproductionprocess described later in detail is performed in Step S04.

When it is determined in Step S02 that the operation mode is set to thephotographing mode, the process moves to Step S10. In Steps S10 and S11,a latest through image of the subject referred to as a so-called liveview is displayed on the image display section 10. Specifically, in StepS10, photographing is performed by at least one of the leftphotographing section 2 a and the right photographing section 2 b of thephotographing section 2, and the image is displayed on the image displaysection 10 in Step S11.

Then in Step S12, the 3D effect determination process described withreference to FIG. 4 is performed. Specifically, it is determined whetherthe 3D effect is effective for the image photographed in Step S10.

When it is determined in Step S12 that the 3D effect is effective, inStep S14, it is displayed on the image display section 10 that a 3Dstill image for an effective stereoscopic display can be photographed.The 3D effect being effective may be outputted by sound. In Step S10,when photographing is performed by both the left photographing section 2a and the right photographing section 2 b, in Step S14, the leftviewpoint image and the right viewpoint image may be simultaneouslydisplayed on the image display section 10 for a 3D display.

Then in Steps S15 and S20, it is determined whether the user inputs aninstruction to photograph a still image or a moving image. When it isdetermined that the instruction to photograph a still image or a movingimage is not inputted, the process returns to Step S01.

When it is determined in Step S15 that the user operates the still imagephotographing switch 4 to input an instruction for a photographingoperation of a 2D still image or a 3D still image, the process moves toStep S16, and the photographing section 2 photographs the 2D still imageor the 3D still image and records the image in the recording section 12.In Step S16, which of the 2D still image and the 3D still image isphotographed and recorded depends on an operation conditionpreliminarily set by the user.

The photographing control section 23 may automatically switch theoperation of the photographing section 2 so as to photograph a 3D stillimage when the 3D effect is effective, and photograph a 2D still imagewith only one of the left photographing section 2 a and the rightphotographing section 2 b when the 3D effect is ineffective. With such aconfiguration, one of the left photographing section 2 a and the rightphotographing section 2 b that is not used for photographing can bestopped when the 3D effect is ineffective, thereby reducing powerconsumption. A recording capacity of the recording section 12 can bealso saved.

When it is determined in Step S20 that the user operates the movingimage photographing switch 5 to input an instruction to start aphotographing operation of a moving image, the process moves to StepS21, and photographing and recording of a 2D moving image are started.

In the photographing operation of the 2D moving image, as shown in Step22, processes in Steps S23 to S28 are repeated until the moving imagephotographing switch 5 is operated again to input an instruction tofinish the photographing operation of the moving image.

Specifically, in Step S23, the 3D effect determination process describedwith reference to FIG. 4 is performed. Specifically, it is determinedwhether a 3D effect of a photographed latest image (frame) is effective.When it is determined in Step S23 that the 3D effect is ineffective, theprocess returns to Step S22.

When it is determined in Step S23 that the 3D effect is effective, inStep S25, it is displayed on the image display section 10 that a 3Dstill image for an effective stereoscopic display can be photographed.

Then, in Step S26, the climax determination process described withreference to FIG. 5 is performed. Specifically, it is determined whetherthe photographed latest image (frame) shows a scene suitable for astereoscopic display. When the climax determination is ineffective inStep S26, the process returns to Step S22.

When the climax determination is effective in Step S26, it is determinedin Step S28 whether a predetermined time has passed since aphotographing time of the previous still image. When the predeterminedtime has passed since the photographing time of the previous stillimage, the photographing section 2 photographs a 3D still image andrecords the image in the recording section 12. When the predeterminedtime has not passed since the photographing time of the previous stillimage, the process returns to Step S22.

The determination in Step S26 is performed to prevent overlaps of 3Dstill images having the same composition by photographing the 3D stillimages at sequential timing. For example, a predetermined time for thedetermination in Step S26 is about ⅕ to 1 second.

The above-described operation is repeated simultaneously with thephotographing and recording operations of the 2D moving image until itis determined in Step S22 that the instruction to finish thephotographing operation of the 2D moving image is inputted. When it isdetermined in Step S22 that the instruction to finish the photographingoperation of the 2D moving image is inputted, the process moves to StepS30, and photographing of the 2D moving image is finished.

In Step S31, it is determined whether a plurality of 3D still images arephotographed in Step S29 during photographing of the present 2D movingimage. When only one 3D still image is photographed, the process movesto Step S35, a photographing time of the 2D moving image andphotographing timing of the 3D still image are connected to record arelationship between the 2D moving image and the 3D still image.

When it is determined in Step S31 that the plurality of 3D still imagesare photographed in Step S29, the process moves to Step S32, and it isdetermined whether there are images among the plurality of 3D stillimages, which are photographed sequentially at a shorter time intervalthan a predetermined time interval. The predetermined interval hereinis, for example, about 3 to 10 seconds.

In Step S32, when there is no 3D still image photographed sequentiallyat a shorter time interval than the predetermined time interval, theprocess moves to Step S35, the photographing time of the 2D moving imageand photographing timing of all the 3D still images are connected torecord a relationship between the 2D moving image and the 3D stillimages.

In Step S32, when there is an image group including a plurality of 3Dstill images photographed sequentially at a shorter time interval thanthe predetermined time interval, the process moves to Step S33, and theimage group is recorded. Then, in Step S34, a 3D still image finallyphotographed in the image group is selected as a representative.

Then in Step S35, the photographing time of the 2D moving image isconnected to photographing timing of all the 3D still images that arenot grouped and photographing timing of the image group to record arelationship between the 2D moving image and the 3D still images.

The above is the description on the operation in the photographing modeof the image pickup apparatus 1 of the present embodiment. Next, withreference to FIG. 8, a reproduction process performed when it isdetermined in Step S03 that the operation mode is set to thereproduction mode will be described.

As shown in Step 50, the reproduction process is performed during aperiod when the operation mode of the image pickup apparatus 1 is set tothe reproduction mode. In the reproduction process, first in Step S51, alist of still images and moving images recorded in the recording section12 is displayed on the image display section 10 as a so-called thumbnaildisplay.

When the user inputs an instruction to select a still image or a movingimage with the operation member, it is determined in Step S53 whetherthe selected image is the still image or the moving image.

When the still image is selected, in Step S54, the selected still imageis enlarged and displayed on the image display section 10. When it isdetermined that the user operates the cross switch 9 or the like toinput an advance instruction (YES in Step S55), it is determined that anext still image or moving image is selected, and the process returns toStep S53. When it is determined that the user operates the zoomoperation switch 8 or the like to input an instruction for a listdisplay (YES in Step S56), the process returns to Step S50.

When it is determined in Step S53 that the user selects the movingimage, in Step S60, the selected moving image is reproduced anddisplayed on the image display section 10.

When the moving image is the 2D moving image and the 3D still imageconnected in Step S35 described above is recorded in the recordingsection 12 during reproduction of the moving image, the processes fromStep S61 to S65 are performed.

In the present embodiment, when photographing timing of the connected 3Dstill image is reached (YES in Step S61) in reproducing the 2D movingimage, the corresponding 3D still image is reproduced and displayed fora predetermined reproduction time on the image display section 10. Thevoice recorded in the 2D moving image is reproduced without beingstopped during the display of the 3D still image and outputted from thespeaker 7. The predetermined reproduction time for reproducing anddisplaying the 3D still image is, for example, about 3 seconds.

After the 3D still image is displayed and the predetermined reproductiontime has passed (YES in Step S64), the 2D moving image is again skippedand reproduced for the predetermined reproduction time and displayed onthe image display section 10.

Next, an actual operation of the image pickup apparatus 1 of the presentembodiment described above will be described below. Hereinafter, forexample, at a photographing time t=0, the moving image photographingswitch 5 is operated to photograph and record the 2D moving image shownin FIG. 9. A left side in FIG. 9 shows changes in scenes of the 2Dmoving image, and shows a state where the image pickup apparatus 1captures a person as a main subject and approaches the person from thephotographing time t=0 to t=2.

In the scene at the photographing time t=1 in FIG. 9, the person's faceis captured at the center of the image, but a distance between the imagepickup apparatus 1 and the person is longer than the suitable value LA.Thus, the result of the 3D effect determination process by the subjectsituation determination section 22 is ineffective.

In the scenes at the photographing time t=2 and thereafter, the person'sface is captured at the center of the image, and the distance betweenthe image pickup apparatus 1 and the person is less than the value LA.Thus, in the scenes at the photographing time t=2 and thereafter, theresult of the 3D effect determination process by the subject situationdetermination section 22 is effective.

In the scene at the photographing time t=3, the size of the person'sface changes and then the change of the person's face stops, and thusthe result of the climax determination process is effective and a 3Dstill image is photographed (Step S29).

In the next scene at the photographing time t=4, the facial expressionof the person is a smile, and thus the result of the climaxdetermination process is effective. However, a predetermined time hasnot passed since the photographing of the previous 3D still image at thephotographing time t=3, and thus the 3D still image is not photographed(NO in Step S28).

In the next scene at the photographing time t=5, the facial expressionof the person is a smile, and the result of the climax determinationprocess is effective. Also, the predetermined time has passed since thephotographing of the previous 3D still image at the photographing timet=3, the 3D still image is photographed (Step S29). In the scene at thephotographing time t=6, the facial expression of the person is not asmile, and the facial expression of the person's face is being changed,and thus the result of the climax determination process is ineffective.

The two 3D still images photographed at the photographing times t=3 andt=5 are sequentially photographed at a shorter time interval than apredetermined time interval, and is thus connected to the 2D movingimage as one image group and recorded in the recording section 12. Then,the 3D still image at the photographing time t=5 finally photographed inthe image group is recorded as a representative.

Next, an operation when the 2D moving image in FIG. 9 described above isreproduced in the reproduction mode will be described with reference toFIG. 10. FIG. 10 shows the display on the image display section 10 whenthe 2D moving image in FIG. 9 is reproduced.

In the reproduction of the 2D moving image, a time before thereproduction time t=2 is not photographing timing of the connected 3Dstill image, and thus the 2D moving image is reproduced and displayed.At the reproduction time t=3, the photographing timing of the connected3D still image is reached, and thus the 3D still image as arepresentative in the image group photographed at this timing isdisplayed on the image display section 10. The 3D still image as therepresentative is the 3D still image photographed at the photographingtime t=5. After the predetermined reproduction time has passed (afterreproduction time t=6), the 2D moving image is again displayed on theimage display section 10.

As described above, the image pickup apparatus 1 of the presentembodiment can automatically photograph a 3D still image in a scenehaving a high stereoscopic display effect and suitable for astereoscopic display in photographing a 2D moving image.

The image pickup apparatus 1 of the present embodiment can automaticallyinsert a display of a 3D still image having a high stereoscopic displayeffect only in a required position suitable for a stereoscopic displayin reproducing the photographed 2D moving image.

Thus, the image pickup apparatus 1 of the present embodiment cansuitably switch between the 2D moving image and the 3D still image toprovide impressive performance expression that can enhance a feeling of“being there” by a stereoscopic display.

A viewer who views a 3D moving image for long hours may have fatigue ordiscomfort of the eyes, but in the present embodiment, the 3D stillimage is displayed in the middle of the 2D moving image, therebypreventing fatigue or discomfort of the eyes of the viewer.

In the present embodiment, the 3D still image is automaticallyphotographed in a scene having a high stereoscopic display effect andsuitable for a stereoscopic display in photographing the 2D movingimage, but the image pickup apparatus 1 may automatically photograph a3D moving image only in a predetermined period in a scene having a highstereoscopic display effect and suitable for a stereoscopic display inphotographing the 2D moving image. Specifically, when the 3D effectdetermination and the climax determination are effective inphotographing the 2D moving image, the image pickup apparatus 1 mayphotograph a 3D moving image of, for example, about 2 to 5 seconds.

In the list display (thumbnail display) in Step S51 in the reproductionprocess, a 3D still image automatically photographed in photographingthe moving image and reduced in size may be used as a reduced-size image(thumbnail) for searching a moving image. With such a configuration, animpressive scene with the 3D effect determination and the climaxdetermination being effective is displayed as the reduced-size image forsearching the moving image, and thus the user can easily find a targetmoving image.

In the above-described embodiment, the image pickup apparatus 1 includesthe pair of left photographing section 2 a and right photographingsection 2 b corresponding to the two viewpoints, but for example, thepresent invention may be applied to a configuration in which two imagepickup apparatuses each having a single photographing section are usedin combination, and the two image pickup apparatuses are substantiallysynchronously operated by wired or wireless communication.

(Second Embodiment)

In the climax determination process shown in FIG. 5 of the firstembodiment, the climax determination is performed based on the subjectand the surrounding state of the subject such as the facial expressionof the person or the volume of surrounding voice, but the climaxdetermination may reflect a photographing intention of a user(photographer) of the image pickup apparatus 1.

Now, as an image pickup apparatus of a second embodiment, an imagepickup apparatus in which such a user's intention is incorporated intoclimax determination will be described with reference to FIGS. 11 and12. The second embodiment is different from the first embodiment only inthe climax determination process. Only the difference will be describedbelow, and the same components as in the first embodiment are denoted bythe same reference numerals and descriptions thereof will be omitted.

As shown in the flowchart in FIG. 11, the climax determination processin the present embodiment is different from the climax determinationprocess in the first embodiment in that Step S89 is added after Nodetermination in Step S86.

In Step S89, it is determined whether the image pickup apparatus 1 ismoved toward the subject and then stopped with respect to the subject.The movement of the image pickup apparatus 1 toward the subject can bedetected, for example, by determining whether a movement vector of animage is radially generated. The movement of the image pickup apparatus1 toward the subject can be also detected based on an output of anacceleration sensor provided in the image pickup apparatus.

In Step S89, when it is determined that the image pickup apparatus 1 ismoved toward the subject and then stopped with respect to the subject,the climax determination is determined to be effective.

In the present embodiment with Step S89 added to the climaxdetermination process, as shown in FIG. 12, when the user 100 pushes andmoves the image pickup apparatus 1 toward the subject 101 and then stopsand holds the image pickup apparatus 1 in a predetermined position, theclimax determination becomes effective, and a 3D image is photographed.

Thus, in the present embodiment, the user's intention can beincorporated into the climax determination, and photographing reflectinguser's performance intention can be performed.

(Third Embodiment)

In the above-described embodiment, the image pickup apparatus 1 includesthe pair of left photographing section 2 a and right photographingsection 2 b corresponding to the two viewpoints, but for example, asshown in FIG. 13, the present invention may be applied to aconfiguration in which one image pickup apparatus 1 b including aphotographing section with a single viewpoint is used, and a subject 101is photographed from a plurality of viewpoints while the image pickupapparatus 1 b is being moved.

A third embodiment in which the present invention is applied to an imagepickup apparatus 1 b including a photographing section with a singleviewpoint will be described with reference to FIGS. 13 to 18. In thepresent embodiment, a user 100 performs photographing while moving theimage pickup apparatus 1 b including the photographing section with thesingle viewpoint from left to right with respect to the subject 101 asshown by an arrow B in FIG. 13 by way of example.

A main routine of operation control of the image pickup apparatus 1 b ofthe present embodiment will be described with reference to theflowcharts in FIGS. 14 and 15. The main routine is started by the imagepickup apparatus 1 b being powered on.

First in Step S101, it is determined whether an instruction to power offthe image pickup apparatus 1 b is inputted. When it is determined thatthe instruction to power off the image pickup apparatus 1 b is inputted,a finish process such as moving a position of a photographing lens to apredetermined position is performed, and then the image pickup apparatus1 b is powered off.

On the other hand, when it is determined that the instruction to poweroff the image pickup apparatus 1 b is not inputted, the process moves toStep S102, and it is determined whether an operation mode of the imagepickup apparatus 1 b is set to a photographing mode.

When it is determined in Step S102 that the operation mode is not set tothe photographing mode, in Step S103, it is determined whether theoperation mode of the image pickup apparatus 1 b is set to areproduction mode. When it is determined in Step S103 that the operationmode is not set to the reproduction mode, the presence of an abnormalityin setting of the operation mode is displayed on the image displaysection, and the process returns to Step S101. When it is determined inStep S03 that the operation mode is set to the reproduction mode, areproduction process is performed in Step S104.

When it is determined in Step S102 that the operation mode is set to thephotographing mode, the process moves to Step S110. In Steps S110 andS111, a latest through image of the subject referred to as a so-calledlive view is displayed on the image display section. Specifically, inStep S110, photographing is performed by the photographing section, andthe image is displayed on the image display section in Step S111.

Then in Step S112, the 3D effect determination process described withreference to FIG. 4 is performed. Specifically, it is determined whetherthe 3D effect is effective for the image photographed in Step S10.

When it is determined in Step S112 that the 3D effect is effective, inStep S114, it is displayed on the image display section that a 3D stillimage for an effective stereoscopic display can be photographed.

Then in Steps S115 and S120, it is determined whether the user inputs aninstruction to photograph a 2D image or a 3D image. When it isdetermined that the instruction to photograph a 2D image or a 3D imageis not inputted, the process returns to Step S101.

In Step S115, when it is determined that the instruction for aphotographing operation of a 2D image is inputted, the process moves toStep S116, and the photographing section photographs the 2D image andrecords the image in the recording section.

When it is determined in Step S120 that an instruction to start thephotographing operation of the 3D image is inputted, the process movesto Step S121, and photographing and recording of a 2D moving image arestarted. Then in Step S122, an image of a frame first obtained is storedas a reference image. The photographing operation of the 2D moving imageis repeated until an instruction to finish the photographing operationis inputted as shown in Step S123.

In the photographing operation of the 2D moving image, first in StepS124, the obtained latest image is compared with the reference image,and it is determined whether a combination of the latest image and thereference image can obtain a 3D image that can be 3D displayed.

In Step S124, it is determined that the 3D image can be photographedwhen the latest image and the reference image are images of the samesubject photographed from different viewpoints with parallax, and thereis no change in the subject.

For example, as shown in FIG. 13, when the subject 101 is a person, andthere is no change in size and facial expression of the face of thesubject 101 in the latest image and the reference image, and a viewpointof the latest image and a viewpoint of the reference image aresubstantially horizontally apart from each other, it is determined thatthe 3D image can be photographed.

The positions of the viewpoint of the latest image and the viewpoint ofthe reference image can be determined, for example, by determining thatthe viewpoint of the latest image and the viewpoint of the referenceimage are substantially horizontally apart from each other when there isno change in vertical position of the face of the subject 101 in thelatest image and the reference image and the position of a backgroundwith respect to the face of the subject 101 laterally moves.

The determination whether the 3D image can be photographed in Step S124may be based on a movement vector of the image pickup apparatus 1 bcalculated from an output of a movement detection device such as anacceleration sensor or an angular acceleration sensor provided in theimage pickup apparatus 1 b other than the above-described comparison ofthe image.

When it is determined in Step S124 that the 3D image can be photographed(YES in Step S125), in Step S126, the latest image and the referenceimage are connected and recorded as a 3D image. Then in Step S127, it isdisplayed on the image display section that the 3D image isphotographed.

On the other hand, when it is determined in Step S124 that the 3D imagecannot be photographed (NO in Step S125), in Step S130, it is determinedwhich of the latest image and the reference image is suitable as animage used for a 3D image.

For example, when the facial expression of the subject 101 changes to asmile in the determination of Step S124, it is determined that thelatest image with the smile is suitable for an image that constitutesthe 3D image, and the present reference image is unsuitable.

When it is determined that the present reference image is unsuitable foran image used for the 3D image (YES in Step S130), in Step S131, thelatest image is stored as a new reference image, and the process returnsto Step S123.

When it is determined that the present reference image is suitable foran image used for the 3D image (NO in Step S130), in Step S133, a 3Dphotographing guide that suggests the user 100 an operation method forphotographing a 3D image is displayed on the image display section.

In Step S133, it is determined which movement of the image pickupapparatus 1 b allows 3D image photographing, for example, from thecomparison result between the reference image and the latest image, andthe direction of the movement is displayed on the image display sectionas a 3D photographing guide. For example, as shown in FIG. 16, when theposition of the face of the subject 101 is moved up with respect to thereference image, the fact is displayed as a 3D photographing guide 111.

The above-described operation is repeated until it is determined in StepS123 that the instruction to finish the photographing operation isinputted. When it is determined in Step S123 that the instruction tofinish the photographing operation is inputted, the process moves toStep S140, and photographing of the 2D moving image is finished.

Next, an operation example of 2D moving image photographing in Step S121and thereafter of the image pickup apparatus 1 b of the presentembodiment as described above will be described.

As shown by a solid line in FIG. 13, when the user 100 first holds theimage pickup apparatus 1 b on the left with respect to the subject 101and inputs an instruction to photograph a 2D moving image, an imagefirst obtained is stored as a reference image (Step S122).

When the user 100 does not move the image pickup apparatus 1 b, as shownin FIG. 17, the photographed latest frame image is the same as thereference image, and thus the 3D image cannot be constituted by acombination of the latest frame image and the reference image (NO inStep S125). Thus, a 3D photographing guide display 110 to instruct tomove the image pickup apparatus 1 b to the right is displayed togetherwith the latest image on the image display section.

When the user 100 moves the image pickup apparatus 1 b substantiallyhorizontally to the right, as shown in FIG. 18, the latest frame imagehas parallax with respect to the reference image, and the 3D image canbe constituted by the combination of the latest frame image and thereference image (YES in Step S125). In this case, the combination of thereference image and the latest image is recorded as a 3D still image inthe recording section (Step S126). Then, it is displayed on the imagedisplay section that the 3D image is photographed (Step S127, referencenumeral 112 in FIG. 18).

As described above, when the image pickup apparatus 1 b of the presentembodiment photographs a 2D moving image while moving with respect tothe subject, and there is a combination that can constitute the 3D stillimage in each frame of the 2D moving image, the combination is recordedas a 3D still image.

Thus, the image pickup apparatus 1 b of the present embodiment canautomatically photograph the 3D still image inserted into the 2D movingimage to provide impressive performance expression that can enhance afeeling of “being there” by a stereoscopic display.

In the image pickup apparatus 1 b of the present embodiment, as areduced-size image (thumbnail) for searching a 2D moving image, one ofthe images that constitute the 3D still image automatically photographedin photographing the 2D moving image and is reduced in size may be used.With such a configuration, an impressive scene in the 2D moving image isdisplayed as a reduced-size image for searching the moving image, andthus the user can easily find a target moving image.

The present invention is not limited to the above-described embodiment,but may be changed within the gist or the idea of the invention readfrom claims and the entire specification, and an image pickup apparatuswith such a change is included in the technical scope of the presentinvention.

For example, the present invention may be applied to electronicequipment having a photographing function such as a recording device, acell phone, a PDA, a personal computer, a game machine, a digital mediaplayer, a television, a GPS, a clock, or the like, not limited to aso-called digital camera as the image pickup apparatus described in theembodiment.

What is claimed is:
 1. An image pickup apparatus comprising: aphotographing section that can photograph a subject from a plurality ofviewpoints with parallax, and can photograph a 2D moving image of thesubject obtained by photographing from at least one of the viewpointsand a 3D image of the subject obtained by photographing from theplurality of the viewpoints; a recording section that records the 2Dmoving image and the 3D image; a subject situation determination sectionthat determines a timing suitable for photographing the 3D image whilephotographing the 2D moving image; and a photographing control sectionthat controls the photographing section so as to photograph the 3D imagewhen the subject situation determination section determines that thetiming is suitable for photographing the 3D image, wherein the subjectsituation determination section calculates a distance between thesubject and the photographing section from a focusing length of thephotographing section, and determines the timing suitable forphotographing the 3D image when the distance between the subject and thephotographing section is less than a predetermined distance.
 2. Theimage pickup apparatus according to claim 1, further comprising a facerecognizing section that recognizes a size of a person's face in the 2Dmoving image by image processing, wherein the subject situationdetermination section calculates the distance between the subject andthe photographing section based on the size of the person's face in the2D moving image and a focal length of the photographing section.
 3. Theimage pickup apparatus according to claim 2, wherein the facerecognizing section can determine a facial expression of the person'sface in the 2D moving image, and the subject situation determinationsection determines the timing suitable for photographing the 3D imagebased on a change in the facial expression of the person's face in the2D moving image.
 4. The image pickup apparatus according to claim 1,further comprising a face recognizing section that recognizes a size ofa person's face in the 2D moving image by image processing, wherein thesubject situation determination section calculates the distance betweenthe subject and the photographing section based on the size of theperson's face in the 2D moving image and a focal length of thephotographing section.
 5. The image pickup apparatus according to claim4, wherein the face recognizing section can determine a facialexpression of the person's face in the 2D moving image, and the subjectsituation determination section determines the timing suitable forphotographing the 3D image based on a change in the facial expression ofthe person's face in the 2D moving image.
 6. An image pickup apparatuscomprising: a photographing section that can photograph a subject from aplurality of viewpoints with parallax, and that can photograph (1) a 2Dmoving image of the subject obtained by photographing from at least oneof the plurality of viewpoints and (2) a 3D image of the subjectobtained by photographing from the plurality of the viewpoints; arecording section that records the 2D moving image in connection with agroup of 3D images including the 3D image; a subject situationdetermination section that determines a timing suitable forphotographing the 3D image while photographing the 2D moving image; anda photographing control section that (1) controls the photographingsection to photograph the 3D image responsive to a determination, by thesubject situation determination section, that the timing is suitable forphotographing the 3D image, and (2) otherwise controls the photographingsection to photograph the 3D image responsive to a determination, by thesubject situation determination section, that the timing is not suitablefor photographing the 3D image, wherein the subject situationdetermination section calculates a distance between the subject and thephotographing section from a focusing length of the photographingsection, and determines the timing suitable for photographing the 3Dimage when the distance between the subject and the photographingsection is less than a predetermined distance.